feat(SDK+backend): Add optional field for the secret/configmap as env vars #12216

DharmitD · 2025-09-04T22:37:53Z

Description of your changes:

Adds support for the optional parameter to kfp.kubernetes.use_secret_as_env() and kfp.kubernetes.use_config_map_as_env() functions, bringing parity with the existing *_as_volume methods.

Python SDK: Added optional: bool = False parameter to both functions
Protobuf: Added optional field to SecretAsEnv and ConfigMapAsEnv messages
Backend: Updated driver to properly handle the optional field in Kubernetes resources
Tests: Added comprehensive test coverage for all optional field scenarios

Behavior

optional=False (default): Secret/ConfigMap must exist (backward compatible)
optional=True: Pod can start even if Secret/ConfigMap is missing
Follows Kubernetes EnvVarSource schema

Example Usage

import kfp.kubernetes as kubernetes

# Secret that must exist (current behavior - backward compatible)
kubernetes.use_secret_as_env(
    task,
    secret_name='required-secret',
    secret_key_to_env={'token': 'API_TOKEN'}
)

# Secret that's optional (new feature)
kubernetes.use_secret_as_env(
    task,
    secret_name='optional-secret',
    secret_key_to_env={'debug_key': 'DEBUG_TOKEN'},
    optional=True  # New parameter
)

This PR includes a cherry-picked commit from @rimolive 's past PR, that consisted of the SDK side changes. I've added backend changes and tests on top of those.

Checklist:

You have signed off your commits
The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

…ings - Regenerate Python protobuf bindings to include the optional field for SecretAsEnv and ConfigMapAsEnv - Update test_volume.py to expect optional: False in secretAsEnv - Include backend driver code that handles optional field (from stash) - All 110 kfp-kubernetes-library-tests now pass - Fixes PR kubeflow#12216 test failures

…ings - Regenerate Python protobuf bindings to include the optional field for SecretAsEnv and ConfigMapAsEnv - Update test_volume.py to expect optional: False in secretAsEnv - Include backend driver code that handles optional field (from stash) - All 110 kfp-kubernetes-library-tests now pass - Fixes PR kubeflow#12216 test failures Signed-off-by: ddalvi <[email protected]>

…ndling - Regenerate Python protobuf bindings to include the optional field for SecretAsEnv and ConfigMapAsEnv - Update test_volume.py to expect optional: False in secretAsEnv - Fix backend driver to only set Optional field when explicitly true (maintains backward compatibility) - Prevents breaking existing backend tests that expect Optional to be nil by default - All 110 kfp-kubernetes-library-tests now pass - All backend driver tests now pass - Fixes PR kubeflow#12216 test failures Signed-off-by: ddalvi <[email protected]>

…ndling - Regenerate Python protobuf bindings to include the optional field for SecretAsEnv and ConfigMapAsEnv - Update test_volume.py to expect optional: False in secretAsEnv - Fix backend driver to properly handle Optional field for all cases: * Not specified: Optional = nil → defaults to false in YAML (backward compatible) * Explicitly false: Optional = &false → optional: false in YAML * Explicitly true: Optional = &true → optional: true in YAML - All 110 kfp-kubernetes-library-tests pass - All backend driver tests pass - Sample pipeline compilation tests pass with correct optional field handling - Should fix KFP Samples test failures related to pod-spec-patch - Fixes PR kubeflow#12216 test failures Signed-off-by: ddalvi <[email protected]>

mprahl · 2025-09-10T14:19:12Z

/assign @HumairAK

HumairAK · 2025-09-10T15:15:59Z

kubernetes_platform/proto/kubernetes_executor_config.proto


    // Name of the ConfigMap.
-    ml_pipelines.TaskInputsSpec.InputParameterSpec config_map_name_parameter = 3;
+    ml_pipelines.TaskInputsSpec.InputParameterSpec config_map_name_parameter = 4;


you should not change the numbers of pre-existing message fields, it will disrupt the wire format for the proto messages, keep the old fields as is and only increment the numbers for the newly added fields

it looks like the one in ConfigMapAsEnv accidentally skipped a number - it is what it is though

oops! Updated to keep the old fields as is, and have incremented numbers for the new "optional" fields.

HumairAK · 2025-09-10T15:21:01Z

backend/src/v2/driver/k8s.go

+			if secretAsEnv.Optional != nil {
+				secretKeySelector.Optional = secretAsEnv.Optional
+			}


can you add a unit test for this codepath?

Done, added tests here.

Signed-off-by: ddalvi <[email protected]>

…iables - Update protobuf definitions and regenerate bindings - Fix backend driver to handle Optional field correctly - Update tests and ensure backward compatibility Signed-off-by: ddalvi <[email protected]>

HumairAK · 2025-09-12T16:28:05Z

/lgtm
/approve

google-oss-prow · 2025-09-12T16:28:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: HumairAK

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~backend/OWNERS~~ [HumairAK]
~~kubernetes_platform/OWNERS~~ [HumairAK]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

… vars (kubeflow#12216) * Add optional field for the secret/configmap as env vars Signed-off-by: ddalvi <[email protected]> * Add tests for the optional field for secret/configmap environment variables - Update protobuf definitions and regenerate bindings - Fix backend driver to handle Optional field correctly - Update tests and ensure backward compatibility Signed-off-by: ddalvi <[email protected]> --------- Signed-off-by: ddalvi <[email protected]> Co-authored-by: Ricardo M. Oliveira <[email protected]>

google-oss-prow bot requested review from droctothorpe and zazulam September 4, 2025 22:37

google-oss-prow bot added the size/L label Sep 4, 2025

DharmitD marked this pull request as draft September 4, 2025 22:51

google-oss-prow bot added the do-not-merge/work-in-progress label Sep 4, 2025

DharmitD force-pushed the apply-pr-11486-changes branch from a5d67de to 10cdb55 Compare September 4, 2025 23:01

DharmitD force-pushed the apply-pr-11486-changes branch from d96ac32 to 3293763 Compare September 4, 2025 23:31

DharmitD force-pushed the apply-pr-11486-changes branch 2 times, most recently from d9c9b5b to a1bcee5 Compare September 6, 2025 00:47

DharmitD force-pushed the apply-pr-11486-changes branch from a1bcee5 to e08a2b9 Compare September 6, 2025 02:41

DharmitD changed the title ~~Add optional field for the secret/configmap as env vars~~ feat(SDK+backend):Add optional field for the secret/configmap as env vars Sep 6, 2025

DharmitD changed the title ~~feat(SDK+backend):Add optional field for the secret/configmap as env vars~~ feat(SDK+backend): Add optional field for the secret/configmap as env vars Sep 6, 2025

DharmitD marked this pull request as ready for review September 8, 2025 15:43

google-oss-prow bot removed the do-not-merge/work-in-progress label Sep 8, 2025

DharmitD force-pushed the apply-pr-11486-changes branch from e08a2b9 to d0b11af Compare September 8, 2025 17:02

google-oss-prow bot assigned HumairAK Sep 10, 2025

HumairAK reviewed Sep 10, 2025

View reviewed changes

DharmitD force-pushed the apply-pr-11486-changes branch from d0b11af to b201949 Compare September 11, 2025 18:19

google-oss-prow bot added size/XL and removed size/L labels Sep 11, 2025

DharmitD force-pushed the apply-pr-11486-changes branch 2 times, most recently from 91d0e48 to 3996e21 Compare September 11, 2025 19:05

rimolive and others added 2 commits September 11, 2025 15:57

Add optional field for the secret/configmap as env vars

e6e07ab

Signed-off-by: ddalvi <[email protected]>

Add tests for the optional field for secret/configmap environment var…

70fc60d

…iables - Update protobuf definitions and regenerate bindings - Fix backend driver to handle Optional field correctly - Update tests and ensure backward compatibility Signed-off-by: ddalvi <[email protected]>

DharmitD force-pushed the apply-pr-11486-changes branch from 3996e21 to 70fc60d Compare September 11, 2025 19:58

google-oss-prow bot added the lgtm label Sep 12, 2025

google-oss-prow bot added the approved label Sep 12, 2025

google-oss-prow bot merged commit 55ec846 into kubeflow:master Sep 12, 2025
93 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(SDK+backend): Add optional field for the secret/configmap as env vars #12216

feat(SDK+backend): Add optional field for the secret/configmap as env vars #12216

Uh oh!

DharmitD commented Sep 4, 2025 •

edited

Loading

Uh oh!

mprahl commented Sep 10, 2025

Uh oh!

HumairAK Sep 10, 2025

Uh oh!

DharmitD Sep 11, 2025

Uh oh!

HumairAK Sep 10, 2025

Uh oh!

DharmitD Sep 11, 2025

Uh oh!

HumairAK commented Sep 12, 2025

Uh oh!

google-oss-prow bot commented Sep 12, 2025

Uh oh!

Uh oh!

Uh oh!

feat(SDK+backend): Add optional field for the secret/configmap as env vars #12216

feat(SDK+backend): Add optional field for the secret/configmap as env vars #12216

Uh oh!

Conversation

DharmitD commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Behavior

Example Usage

Uh oh!

mprahl commented Sep 10, 2025

Uh oh!

HumairAK Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

DharmitD Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

HumairAK Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

DharmitD Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

HumairAK commented Sep 12, 2025

Uh oh!

google-oss-prow bot commented Sep 12, 2025

Uh oh!

Uh oh!

Uh oh!

DharmitD commented Sep 4, 2025 •

edited

Loading